Towards an Integrated Cognitive Architecture for Modeling and Recognizing User Affect

We outline the cognitive model CASS (Cognitive–Affective State System). As the name suggests it is a cognitive model that also takes human affect into account. CASS combines Dynamic Bayesian Networks (DBNs) and an ACT-R model. The DBN model (R-BARS, the Rensselaer Bayesian Affect Recognition System) determines the user’s most likely affective states using both current and stored sensory data. The affective–cognitive model integrates R-BARS with ACT-R to play two roles: (1) the use of model tracing to determine the impact of affective state on cognitive processing, and (2) linking changes in affective state to changes in the value of ACT-R’s parameters so as to directly generate (i.e., predict) the influence of affect on cognition. The cognitive implications of the user’s affective state are determined by analyzing the deviation of user behavior from the optimal path determined by the model. Affective state can influence the users’ cognitive processing capabilities and hence their productivity. The goal of this research is to develop methods to timely and efficiently recognize certain negative user affective states, model the influence that these affective states have on cognition and behavior, and provide the most appropriate intervention in a timely manner to return the user to his/her productive state. We see this work as taking steps towards a more distal goal of developing an integrated architecture of affect and cognition. There are four challenges facing this initiative. (1) User’s affect develops over time, and its expressions vary significantly with individual and context. Transient changes in a user’s affective state cannot be inferred from snapshots of behavior independent of the individual and context; rather, sensory observation must be integrated over time with individual difference (i.e., information on how this individual differs from the average individual) and contextual data. (2) Affective state observations from a given sensory source is ambiguous, uncertain, and incomplete. (3) The influence of cognition on affective state and vice versa is not well understood, (cf., Frijda, Manstead, & Bem, 2000, for an overview of the current state-of-the-art). (4) Interventions to improve user performance must be rendered in a timely and effective manner. Our proposal contrasts with the current state-of-the-art in augmented cognition as well as in affect-based augmentation. The former assume normative performance and fail to adapt to the user’s current emotion state. The latter tends to have low-to-no cognitive fidelity, failing to understand the cognitive activities that lead to the observed user state. (But see Hudlicka, 2003b for an overview of some recent approaches and developments in these fields.) In contrast, our framework addresses both sets of challenges. The proposed framework consists of five major parts: (a) data sensing, (b) user affective modeling, (c) user cognitive modeling, (d) an integrated affective-cognitive model, and (e) a probabilistic user assistance model. Data sensing noninvasively acquires various visual (eyelid movements, gaze, facial expressions, hand gesture, and head gesture), physiological (e.g., hand force and pressure, temperature, heart rate, Galvanic Skin Response), and behavioral (e.g., mouse movement, clicks, current point-of-gaze, and performance) data. Based on Dynamic Bayesian Networks (DBNs), the Rensselaer Bayesian Affect Recognition System (RBARS) (Li & Ji, 2003) determines the user’s most likely affective states using both current and stored sensory data. The model has four components. The context component represents information about environmental factors such as time of day, temperature, working condition, and type of work that may influence the user’s affective state. The affective state component represents the emotional states that the system can infer. Target affective states include fatigue, confusion, frustration, fear, sadness, and anger. The profile component may include age, experience, skill level, personal health, sleep history, etc. The profile component enables us to adapt the user affect model to individual differences between users. The observation component integrates the current data obtained from data sensing with the longitudinal record of data sensing collected during the current session. These four components enable R-BARS to infer the user’s affective state. Based on ACT-R (Anderson & Lebiere, 1998), the user cognitive model can perform the task using the same software interface as the human user. For our current purposes, the more important goal is “model tracing”; namely, the step-by-step tracing of human performance in real-time. Although model tracing in real-time has been repeatedly demonstrated at the 10-s level of analysis (Anderson, 2002), behavior at the 100-ms level, such as point-of-gaze, is viewed as non-deterministic. To better constrain our cognitive interpretations of human behavior at the 100-ms level, we will process the behavioral data sensing measures (discussed above) through a separate DBN. The heart of the proposal is our affective-cognitive model. This model integrates R-BARS with ACT-R to play two roles: (1) the use of model tracing to determine the impact of affective state on cognitive processing, and (2) linking changes in affective state to changes in the value of ACT-R’s parameters so as to directly generate (i.e., predict) the influence of affect on cognition. The cognitive implications of the user’s affective state are determined by analyzing the deviation of user behavior from the optimal path determined by the model. Model tracing aligns model behavior with human behavior. During model tracing, sequences of productions are found (and fired) that suffice to produce the sequence of behavioral data obtained from the user. The difference between expected and obtained behavior will be interpreted in terms of the influence of affective state on cognition and behavior. Towards the second role, ACT-R parameters need be identified that link user affect to cognition. For example, candidate parameters include noise in memory activation and noise in production choice. As activation noise is increased from its default value the probability of retrieving the correct memory decreases and the probability of retrieving an incorrect memory increases. Concomitantly, the time to retrieve a correct memory should tend to increase. Such missed or delayed retrievals would have a subtle impact on the behavior of a cognitive system performing a complex cognitive task. Varying noise in production choice from its default setting has been considered as mimicking the changes in arousal that result in the Yerkes-Dodson inverted-U performance curve (Belavkin, 2001, 2003). As production choice noise is increased from the default, an increasing wide variety of actions are given in response to the same environmental stimuli. Alternatively, as production choice noise is decreased from the default, the system soon develops tunnel vision in that it tends to give a stereotypic response to a given set of stimuli rather than finding the response that is most adaptive. Low levels of arousal, or noise, produce behavior reminiscent of fatigue, whereas high level of arousal and noise yield behaviors reminiscent of confusion. For more general thoughts on the effects of affect on cognitive architectures see Ritter, 1993. Hence, in combining R-BARS with ACT-R our proximal goal is to mimic the effect of affect by identifying lowlevel parameters of the ACT-R architecture that, when varied, mimic the cognitive and behavioral consequences of affective state. Note that the low-level parameters of ACT-R constitute an activation model, and, therefore, influence the model in a different way than parameters in other current models, e.g. Hudlicka (2003a). Beyond this goal, our distal goal entails developing an architecture of cognition whose models will spontaneously develop changes in affective state (with all of the cognitive and behavioral consequences that this implies) as a result of their experience in a given task environment. For example, models that are subjected to high workload demands will develop stress and those that are kept running for a prolonged period of time will develop fatigue. Our goals for this effort are profound and challenging. Even our proximal goal will take more than a few years to achieve. However, a partially validated integrated affective-cognitive model would be an important product in its own right, and an important step forward in our understanding of the relationship between cognition and affect. Indeed, even a partially validated form of our proximal goal would be powerful and useful in predicting the types of interventions that are most likely to assist the user. Given the knowledge of the user’s affective state and its likely causes, and the user’s desired state for productive performance, the user assistance module probabilistically determines the most appropriate user augmentation and its application timing. Information-theoretic criteria will be used to assess each assistance/intervention dynamically in order to maximize the likelihood of returning the user to a productive state while minimizing any potential adverse effects of the assistance. References Anderson, J. R. 2002. Spanning seven orders of magnitude: A challenge for cognitive modeling. Cognitive Science,